AdaLip: An Adaptive Learning Rate Method per Layer for Stochastic Optimization

Abstract Various works have been published around the optimization of Neural Networks that emphasize significance learning rate. In this study we analyze need for a different treatment each layer and how affects training. We propose novel technique, called AdaLip, utilizes an estimation Lipschitz constant gradients in order to construct adaptive rate per can work on top already existing optimizers, like SGD or Adam. A detailed experimental framework was used prove usefulness optimizer three benchmark datasets. It showed AdaLip improves training performance convergence speed, but also made process more robust selection initial global